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This paper is a practical guide for states planning 
*o implement a review panel to enhance f£le maintenance in a career 
information system. It also describes successful methods employed in 
established review panels in California, 'Alaska, Washington,' 
Massachusetts, Nebraska, Michigan, and Oregon. The first tiro brief 
sections introduce the purpose of this publication and address the 
.purpose of a review panel, namely to supplement standard data 
sources, verify information in files and identify possible variations 
in local areas, allow community to react, and promote public 
awareness of the system. In the third section steps and procedures in 
carrying out the review panel process ar$ explained, including (1) . 
preparation, (2) scheduling (determining order in which clusters or 
occupations are scheduled to leave office for, review), (3) selection 
of review panel members, (4) telephoning of prospective reviewers, 
(5) record keeping (methods, assigning /numbers) , (6) mailing 
procedure, (7y recording costs, and (8> continuing the process. 
Section u discusses three necessary considerations for implementing a 
review panel — development of materials, \staffing, and budget. Section 
5 lists nine technigues employed by staff of one career information 
system to improve response rate. Five otl\er improvements are also 
suggested. Appendixes include sample review panel schedule, telephone 
conversation, . mailing list, and budget form. ( YLB) 
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PREFACE 

This paper is designed to be a practical guide for states planning to 
implement, a review panel to enhance file maintenance in a career information 
system. It is also a description of the successful methods employed in 
established review panels. Its intent Is. to provide direction to those 
states planning on implementing a review panel. The paper describes the 
experiences of seven states currently operating review panels . 

Staff members from six career information systems described the review 
panel process in their state. Special . thanks is |due Ann Dragavon, EUREKA, 
the California Career Information System; Kipp Drummond, Alaska Career 
Information System; Bert Palmer, Washington Occupational Irifcirmation Service; 

Robert Perello, Massachusetts Occupational Information Systems Janet Rife, 

i 

Nebraska Career Information System; and Jean Swanson, Michigan Occupational 
Information System, for their cbntributiorfs^through our . teleprnone conver- 
sations. 

The review and discussion provided by Bruce McKinlay, Michael Valliere, 

Frances Miller and Maggie Burton all of 1 the Oregon Career Information System, 

is appreciated. Gayle Asburry also deserves recognition for bsr patience 

and excellent typing ability. 

I wish to also acknowledge Les Janis ,/ Georgia Career Information System; 

Dave Porter, Idaho Career Information System; arid John Post, Alaska Department 

of Labor, for their critique of the paper- 
Valerie Lodewick of the National Occupational Information Coordinating 

Committee and Bob .Alexander of the National Governors' Association also 

reviewed the paper. 

. Support for this paper was provided, by the Fund for the Improvement of 

Post-secondary Education (FIPSE) and the National Occupational Information 

Coordinating Committee (NOICC) . 



INTRODUCTION 



There are many agencies produc±ng occupational ar; 
every state. Student, counselors, job searchers and 
career decision making, however, rarely have trie time 
or analyze these data. The need fox an effective s^st^- 
disseminating career planning information is the prlmar 
statewide career information systems,- 

These systems are based on information developed - 
sources, supplemented by informed opinions of Jcnowled; ^ 
method of acquiring informed opinion, employed by the 
units of several states, is known as the review panel 
of members of local agencies and the community who .ar- 
ticular occupation or training program. 'Reviewers' -.rt 
an occupational description for any erroneous or out 
to answer a questionnaire covering specific occupatirx 

The purpose of this paper is to describe the st- 
ations for implementing a review panel. Seven start ... 
terns have review panels currently in operation; ther - 
collected by telephone interview. Variations among :~: 
review panels show that the process may be adapted t 
ysis needs of any system collecting and delivering c 
follows is the rationale for doing a review panel, r 
most states, and factors that need consideration wh. 
taking place. 



j.*-scz:3csl' data in 
- -« i; , es ted in 
4ki.lls : rra collect 
:iir gsrrisring and 
-aa=g rn- ±or creating 

isr^ng data 
i 'Contacts. One 
-latiom analysis 
= paz:el is composed 
liar -with a par- 
ked t:o examine 
1 {formation and 
d training topics. 
•;cj-asad consider- 
zr: iJir^ormation sys- 
mces have been 
*t€£ systems 1 
inf3"trmation anal- 
■ information. What 
ires common to 
:nlementation is 
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PURPOSE DP ,: REVIEW PANEL 



Ie?velopment and maintenance of cscssir information files require con- 
■ir. ..jus identif ication, collection and g : : i Inatior rf existing data sources. 

*'^ts *may include publications, unpri:Idi£ked Jn,*r.i. y newspaper articles 
:rn. .ivsriisements, and contacts with: ~BopLa cr:irrv_ -cgeable about an occu- 
:.iat- :n: .zr training program. To a larse e^rerr :;mnation about certain 
rzzc. nirional items, such as current enrol nymert. is available through stan- 
dar nai>lished sources. In determining otner . ^cnpsrional characteristics 
rac - s working conditions, however, ±t is .tte ess^ry to consider alternative 

such as informed opinions, because frr mal sources are incomplete. 
Jw:. e-quently, a primary reason for having a :^vi£w panel is to supplement 
st~z iard data sources. 

The review panel process is an even isrrre valuable asset in increasing 
the accuracy of information in the files. As a verification procedure the 
re5v±£iw panel helps to improve the quality jf information in the files, but 
daes not suggest that reviewer's comments are considered to be factual with- 
out being confirmed* Instead, information received through returned ques- 
tir Tnaires is analyzed along with a nurr?er of other formal and informal in- 
formation sources. 

-Review panel members also can pror±de names of other people to contact 
and leads to new data sources, Althou^z: requests to reviewers sometimes 
result in the acquisition of data substantiated reports, the review panel 
is not intended as a data collection survey nor is the panel a scientifically 
drawn sample. Rather than generating hh% ? data, it is a means of verifying 
existing informatiq£ in the files and identifying possible variations in 
local areas. 

The review panel also allows members of the community the opportunity 
to express their point of view and to re: *. to the accuracy of the occupa- 
tional description. Thereby, analysts cat :rovide information containing 
firsthand recommendations for preparing tr 3nter an occupation. Finally, 
by providing some visibility for the systen,\the review panel is a means of 
promoting public awareness of the system. 
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JTEPS AND PROCEDUPvES 



Staff assigned to carr y out the review pane„ process are exp^zted to 
complete a mmber of task- requiring attention t-. detail, organization-, 
and deadlines The desirad outcome is the tiimelir acquisition of ^sg in- 
formation^ F-'-cedures thar encourage maximum response to mailed —.lestion- 
naires azd t^smooth integration of that information into the ocmpational 
review proce^ are used. The tasks follow. 

I. Prenarii cic n 

The fi~ t step for review panel staff at a career information system 
is to ob-a±:. 5 copy of the specific occupational information to send to 
reviewers > _ rn~e states use a computer printout of the occupational infor- 
mation, ±nc__.aing localized areas (if applicable). Using the computer print- 
out assures ihat the most recent information will be sent to reviewers. 
Other state use copies of the printed version; however, any current updates 
entered in: the computer after publication would not be reflected in that 
copy. 

Before engaging in any further steps, familiarizing oneself with the 
material Is important to be able to communicate intelligently with prospec- 
tive review panel members. The list of employers in the 'Work Setting 1 sec- 
tion is a particularly good place to develop an understanding for the types 
of employers that hire workers in a particular occupation, 

II. Scheduling 

Information received from review panel members is- integrated into the 
review cycle, the periodic updating of information in the files. Therefore, 
work on the review panel should begin about 4-6 weeks prior to the beginning 
of the review cycle. This allows enough time to receive some returns, with- 
out being so far ahead that the information obtained is outdated by the time 
it is used. (See Appendix A for sample schedule). 



/ 

/ 
/ 
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The following are the criteria used to determine the order in which 
clusters or occupations are scheduled tn leave the office for review: 

A. Seasonality for some occupations. For exanrple: 

— Sending out quest ionnnires for the agricultural occupations 
in the spring or late suimner may produce a low response rate 
since those are the b:^sy times of the year for farmers, 

" filing the construction cluster mid-wirster could produce a 
'surplus' response to xh outlook question because that is 
the season in which ninny layoffs occur, 

B. Sequence in which the occupational clusters were reviewed during 
the previous review cycL^ t 

C. Review cycle assignments among analyses. Clusters need to be 
scheduled to equally distribute the mailings and returns to be 
most helpful to the anal^srs. 

Some states have elected tc review fewer than, the total number of oc- 
cupations during a year. In addition to the above criteria* they are likely 
to consider some other factors. California and Michigan have adop ted the 
criteria below for scheduling occupations to review with their next review 
panel: 

A. Occupations/clusters excluded from the" last review cycle, 
B* Occupations that received considerable criticism or comments 
from users, 

C. Occupations/clusters having evidence of significant change 
through reports or newspaper articles, etc. (e.g. expected 
outlook change for a particular industry or new legislation 
affecting licensing requirements) , 

D. New occupations. 

III. Selection of Review Panel Members 

While developing and localizing career information a state system iden- 
tifies many data-producing agencies. During the review cycle, the periodic 
review in which the information is maintained, it is essential to continue 
the flow of information from these agencies and to locate other knowledge- 
able sources as well. This flow is accomplished throught staff efforts 
to establish and maintain professional relationships with contacts in data- 
producing agencies and with informed sources in the community. At the • 
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-ame time that contacts are being initiated, the staff can develop some 
zr the original review panel members. For example, a researcher at the 

~zate Department- of Education contacted about a general salary survey the 
-i^ncy produces could also be asked to review the description and answer 
z questionnaire for Elementary and Secondary Teachers. Others selected 
to be reviewers should be familiar with the occupation by virtue of per- 
sonal experience or position. Reviewers might include — 

A. An employer of people in the occupation (e.g., owner, personnel 
manager, supervisor), 

B. A worker in the occupation, 

C. An Employment Service placement specialist familiar " with the 
occupation, 

D. A Labor Market Analyst, 

E. Someone responsibly for preparing people for the occupation 
(e.g., college professor or placement officer, vocational 
instructor, apprenticeship representative), if applicable, 

F. Anyone with specia^L expertise about the occupation, although 
not necessarily working in the occupation at the time (e.g., 
union business agent, member of a licensing board). 

Members of the review panel should have varied involvements with the 
occupation, in order to contribute different perspectives. Reviewers some- 
times contradict each other, depending on their relationship to the occu- 
pation, For example, an educator may report much different outlook and 
wage information than an employer or employee. However, accurate infor- 
mation can be insured only after examining various viewpoints. While up- 
dating occupational information, it is the analyst's responsibility to 
be aware' 06 the reviewers 1 relationship to the occupation and to judge the 
responses accord ipgly. Disagreement among the reviewers requires that the 
analyst search for factors contributing to the discrepancy. Clues can be 
sought in formal sources such' as economic reports and from people who would 
have firsthand knowledge about the occupation. 

After one periodic review of occupations has been completed, the occu- 
pational file folder is a good place to start looking for names of people 
to contact. It should contain telephone interview forms from the previous 
occupational review cycle, on which the analyst indicated that a specific 

i 

• ' ' \ ■ ' 
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person was especially helpful or gave a -erbal commitment to participate in 
the next review panel. Newspaper clippings, articles from local periodicals 
or brochures describing an organization or a program may also be filed in 
the folder-. They sometime cite names that can be used as potential contacts. 
Other common sources for prospective reviewers include: 

A. Directories of Manufacturers (Published by agencies such as the 
State Department of Economic Development) , 

B. Directories of unions or union membership lists, 

C. Association membership lists, 

D. Refer ences from other reviewers 

E. Newspaper articles and classified ads, 

F. The yellow pagjes, 

G. State and federal government telephone directories, 

H. Licensing boards, 

I. £a£alogs of training institutions such as community colleges, 
four-year schools and the. State Apprenticeship Agency, 

J. File of people users can contact to learn more about an occu- 
pation (e.g., "VISIT" file), j 
K. Personal acquaintances of staff members, 1 . 

L. Contacts made by User Services or Information Analysis staff, 
M. Chambers of Commerce. 

Presently, the number of reviewers per occupation varies from three in 
some states to fifteen in others, with five being most common among the states 
conducting review .panels . Adequacy of data sources for the topics being 
reviewed and localization of the information are determinants in deciding 
hou many questionnaires and occupational descriptions to send for each oc- 
cupation. Of course, high quality information is more likely, the more the 
information is reviewed. 

It is desirable to get representation from various geographic areas 
throughout the state, recognizing that occupational aspects, such as wages 
and outlook, vary accordingly. That is , distributing descriptions and 
questionnaires to different areas is beneficial in ascertaining these local 
variations. Determining which areas to represent ' largely depends on the 
employment distribution- for a specif ic 'occupation, . Employment for some oc- 
cupations is fairly evenly distributed with population; therefore review 
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panel members would be apportioned "likewise. " Other occupations are con- 
centrated in one or two areas, and consequently most or all reviewers would 
also be located in those areas. 

IV. Telephoning 

Before mailing questionnaires and occupational descriptions, most state 
systems telephone prospective reviewers to request that they participate. 
The review panel staff member identifies who is calling, where he or she is 
calling from, what a career information system does, and asks if the person 
being contacted would mind reviewing an occupational description and filling . 
out a short questionnaire, (See Appendix B for sample telephone conversation). 
Spelling of the person's name, their street address, and zip code are veri- 
fied if the person agrees to be on the review panel. 

Often this step is most time consuming and can be expensive. It may 
be eliminated altogether > or reduced by excluding those reviewers who have 
previously been contacted either by an analyst during the review cycle, or 
as a review panel member during the preceding year. Most states limit their 
telephoning to those potential reviewers with whom they have had no prior 
contact. California and Washington reported that telephoning is excluded 
from their procedures. However, states that include telephoning as part of 
their process have indicated higher response rates, 

V. Record Keeping 

A. Methods 

\ After determining who the reviewers will be, the review panel staff is / 
responsible for keeping account of the pertinent details. There are several 
ways of keeping records, but the following items are appropriate for all : ^ 
a code number for each reviewer; the reviewer's name; the agency name, address 
and phone number; reviewer's title; the localized area sent (if applicable); 
the date sent; and date returned. Other information, such as whether the 
questionnaire was returned undelivered or the reviewer requested to be 
deleted from the review process the following year, can also be recorded. 

Several review panels catalogue ^the above details by using index cards 
to classify reviewers by occupation or a combination of index cards (for name, 
address, etc.) and mailing lists (for dates sent and receivedy~deletes, etc.) 
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Experience in. Oregon with index cards > mailing lists, and a combination of 
the two has resulted in the exclusive. use of a mailing list, (See Appendix 
C for sample mailing list,) Washington's approach to record keeping is 
distinctly different from the above methods. They store the reviewer's name, 
agency, address,^ and phone number on the computer, and use a printout to in- - 
dicate dates 3ent and returned, 

B. Assigning Numbers 

Regardless of thie system devised for record keeping, a way of deter- 
mining those wfio have returned their materials is useful. One method of 
doing so. involves assigning code numbers by using the occupational number as 
the first four digits, followed by an individual serial number (01, 02, etc.) 
and a number assigned to the area of the state (whether localized or not) to 
which the printout and/or questionnaire is being sent. When noted on the 
questionnaire, this number provides a means of recognizing who responded and 
which occupational folder, the return should be filed in. It is also a good u 
reference for 'the analyst to use 'to identify 'the area of the state.):ef lected 
in the answers. Example: "* ^ 



Librarians 



Areas in Oregon 



VI. 



Occupation 
Number 


Reviewer 
Code 


. Area 
Code 






8492 


01 


06 


06 = 


The Coast ' ■■ * . 


8492 


02 


15 » 


15 = 


Southern : 0regon 


8492 


03 . 


26 


26 = 


Portland Metropolitan 
Area 


Mailing Procedure 









Using the mailing list, index cards or computer printout, the cover 
letters and envelopes are^addressed . The occupational description and prepa- 
ration statement are xeroxed and copies of j the appropriate localized area are 
:made for each reviewer, if that's applicable, A review panel packet should 
include: 

A. Cover letter - personalized by typing in the. date and person's name 
and signed by the review panel staff, 
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B. Questionnaire - labeled with the individual code number assigned 
to a reviewer 'and^ attached to: 

C. A copy of the occupational printout, 

D. A self-addressed envelope. Options include business return enve- 
lopes,,,, and stamped or unstamped self-addressed envelopes, (In 
Massachusetts, it was observed that paying the return postage sig- 
nificantly increased the response rate when the reviewer was self- 
employed or employed by a small organization, but .made little dif- 
ference for large organizations). 

E. A ^rqchure or circular providing some specifics about a career 
information system, how it is funded and who uses it is also 
enclosed with the materials by Alaska and Nebraska. ■ 

After being packaged, the envelope is weighed, stamped and mailed,* 
Postage costs for each occupation are recorded. 

VII. Recording Costs 

„ t To help the manager of the Review Panel budget determine allocation of 
funds and cost/benefit factors of the process, records of the following data 
are kept on the "keview Panel Budget Form" for each cluster: 

.A. The cluster number and date the last occupation was mailed, 

B. ;The number of xeroxed copies madej of the description, 

C. The number of questionnaires and/or cover letters duplicated 
(the total number may be recorded on one cluster's budget form 
as copy orders are filled) , 

; D. The number of supplies, N i,e. envelopes and return envelopes, 
E. The amount spent on Dostage for the original mailing, as well, as 
the number of returns received (if business return envelopes are 
enclosed) since the laaJ budget form was filled out,, 
. F. The number of hours the review panel staff worked on the cluster. 
(See Appendix D for sample budget form.) 

VIII, Continuing the Process 

. The on-going operation of the review panel requires basically continuing 

the process in the same manner as the year before, i.e. following steps I 

f 

through VII. When selecting reviewers, however, the review panel staff should 
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check the mailing list (or other method of recording names) from the previous 
year for reviewers who returned their questionnaire and agreed to continue 
reviewing occupations. These are the people who usually are interested in 
participating and are most likely to return the questionnaire again. However 
in order to obtain a variety of opinions it is preferable to drop their names 
from the list after two or three years. Exceptions include people whom the 
analyst believes are invaluable sources of information or who provide infor- 
mation which may be unobtainable elsewhere (e.g., the only apprenticeship 
field representative for a geographic area) . They may remain as reviewers 
indef intely . \ 

If the mailing lists are used in keeping records, they can be reused 
by crossing off reviewers who had not returned the questionnaire or re- 
quested not to participate in the future. New names may then be added to 
the bottom of the list. Reusing mailing lists saves retyping time but tend,s 
to become disorganized after a couple of years. 



CONSIDERATIONS FOR IMPLEMENTING A REVIEW PANEL 

Before proceeding with any project . there are always certain issues that 
need evaluation. A review panel requires Someone to develop materials and 
allocate staff time and money. Therefore, these subjects are discussed in 
the following paragraphs. 

I. Development of Materials 

Developing a good cov/ar letter and questionnaire is essential to the 
review panel process* 

. The cover letter is important because it is an opportunity to/ promote 
public awareness of the system and generally shapes a reviewer's first, 
impression. It should briefly explain the concept of the system and clearly 
define what it is we are asking of reviewers. 

The questionnaire's purpose is to inquire about the accuracy of infor- 
mation in the career information system files, to supplement standard data 
sources, and to seek information that would not be obtainable through other 
sources, Questionnaire development begins with a meeting of the information 
analysis staff to discuss and decide upon occupational topics that are high 
priority for' review in the coming year. Topics such .as wages , outlook and 
hiring practices are always reviewed^ because of their ever changing nature. 
Keeping in mind that a short questionnaire does not take much of / the reviewer' 
time and is more likely to be returned, other possibilities can /include: 

m f ' ' 

A. Questions dealing wi£h the accuracy and completeness o^E the occu- 
pational description -<and preparation statements and -» tljieir. com- 
ponents such as commonly used and-^^er^a^ii^^ee^fta^ 
aptitudes, work setting, types of employers, and skills, 

B. Questions concerning the strengthening of. certain ar,ias, such as 
related occupations or tips by asking about transferable skills, 
advancement, and promotional opportunities*, / 
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I.. Questions seeking new information to add, such as fringe benefits, 

and .helpful information for minorities and the handicapped, 
Z% A final question inviting continued participation in the review 
panel process is helpful in determining those who are likely to 
respond the following year. 
On the other hand, topics a reviewer probably would not be able to 
answer (e.g., total employment in the state), or topics for which there is 
one very reliable source (e.g., licensing requirements) should be avoided. 

All questions should be general in order to apply to all respondents; 
that is professional, technical, clerical, and trade occupations, etc. 
Considering the diverse audience asked to reply, readability must be a con- 
cern when formulating questions. Wording should be simple and unambiguous. 
Avoiding use of technical language and terms, such as journeyman, used by 
specific occupational groups is important. 

The order in which questions are asked also has an impact on the way 
in which a questionnaire is answered. Asking questions about sometimes 
sensitive subjects such as wages should be delayed until other questions 
can create a favorable attitude. Inquiring about wages, first often causes 
resistance to answering that question, and consequently the entire ques- 
tionnaire. Opening the survey with a request for hiring practices infor- 
• mation has also caused problems. That is, a first question asking about ' 
qualifications, education, or training required when hiring, someone has 
caused some self-employed prof essionals (e.g. \ chiropractors) to respond 
with, information about their assistants instead pf about themselves. Thus, 
much thought must be given to the introductory question. 

Reference to Chapter 6 of Research In Education 1 can be .a useful tool 
in designing a questionnaire. A, review of the.'draft questionnaire by non- 
information analysis staff is also often helpiul. 

Enough time needs to be allowed for design of the letter and question- 
naire and for printing. At. the same time, copies of mailing lists and/or 
index cards, envelopes, budget sheets, and any other supplies should be 
ordered. y ' 

^est, John W., Research In Education . 3rd Ed., (Englewood Cliffs 
New Jersey: Prentice-Hall, 1977). ' ' " 
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II. Staffi=s . / ■ 

f 

There ^e at least two factors which must be considered when planning 
Co staff tz.:3 review panel process effectively: L) the type of personnel to 
employ, anc 2) the number of staff needed (determined by the amount of timfe 
necessary to complete the project). Possibilities for the kind of staff to . 
use and estimates of the time a review panel requires are discussed feelow; 

Options for review pane! staffing include hiring part-time temporary 
employees, auch as students; involving permanent clerical staff employees; 
and incorporating the tasks into an analyses work schedule. Initiating 
an maintaining the review panel process requires the ability to select 
appropriate reviewers; communicate effectively by telephone; type names, 
addresses, and telephone numbers accurately; and keep organized records. 
Soke states prefer analysts to perform at least the first two of the above 
duti^, but all tasks have successfully been accomplished in Oregon by students 
or clerical staff, with coordination by an analyst, 

Time required to complete all steps of the review panel process for one 
occupation ,yary depending upon availability of reviewer names for the occu- 
pation, the eas§ or difficulty experienced in contacting reviewers by phone 
and the staff -persons t f '» typing speed. In Oregon it takes approximately three 
hours to complete one, occupation when mailing twelve to fifteen questionnaires 
per occupation. At that rate 250 occupations could be accomplished in about 
twenty hours per .week, over nine- months, ' ' 

Of course the manner of staffing and number employed, which currently 
varies from state to state, is subject to available resources as well as 
the size of the effort being undertaken. 

III. Budget 

Determining casts for planning ^purposes Is always an important but diffi- 
cult iiask. Using 3,000 reviewers an example (that is twelve reviewers 1 
for 25D occupations or nine reviewers for 350 occupations) the following 
figures may be estimated: 
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Supply 

3000 cover letters (@ ,03 per copy) 
3000 questionnaire^ (@ .03 per copy) 
6000 2-page copies of the occupational 

information (@ ,05/copy) xeroxed 
3000 postage (@\ . 15/packet) 
3000 initial cost of printing business 

return envelopes 

I 

3000 replies f ron| business return envelopes 
(assuming art average 65% return rate 
@ 18.50/return) NOTE : If the return, 
rate is increased to over 80% it becomes 
less expensive to use self-stamped return 
envelopes. 

TOTAL SUPPLIES, 



Dollar Amount 

$ 90 
90 
300 

450 
85 

360 1 



$1,375. or $5.50/occupatic 



Other expenses /consist of staff time and telephone bills, which vary 
more than* supplies y Applying the approximation of time required pef occu- 
pation in the •staffing 1 section on page 13 .to the hourly wage of the review 
panel personnel csfn provide a calculation of staff time expenses* That is, 
completion of one occupation in thre$ hours by a student or clerical assistant 
being paid $3.50/hour amounts to $10. 50/occupation in staff time. 

Telephone charges depend upon the type of telephone system used and the 
number of calls made .per occupation, . An average time per call is about five 
minutes, which would' cost $1,25 using the WATS line system at $.25 per minute. 
Telephoning will be most expensive during the first year oi ther "review cycle 
since most reviewers would have had no prior contact. In the following years, 
calling can be reduced by excluding those Who haVe been contacted jpreviou'sly 
as. a member of the review panel or by an analyst during the last review cycle. 

Adding supplies and staff time but "excluding* telephone use, the review 
panel process will cost approximately $16/occupation, \' 
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"T TECHNIQUES FOR IMPROVING THE RESPONSE RATE 

• Seven states presently are conducting review panels. Most states report 
about a 65 percent response rate and have offered suggestions for improving it 

■The following are techniques employed by the staff of one career infor- 
mation system when working with its review panel. They appear to have im- 
proved the response rate: 

A. Printing the questionnaire on colored paper to make it distinctive 
and less likely to be misplaced, 

B. Including a return envelope with the questionnaire to make it 
easier for the respondent to return the questionnaire and ensure 
the accuracy of the return address, 

C. Signing the cover letters. Using blue ink, or any — ^ r other 
than black, make's the signature a more noticeably -3En3onalized 
one, • . 

D. Addressing the letters and envelopes to a person's name rather 
than to a person's title (e.g., manager, librarian) or. to an 
organization." This also personalizes the request, 

E. Clarifying questions by providing examples of the type ofinfor- 
mation being sought to make answering easier, 

F. Asking closed form or multiple choice questions, allowing for 
an 'other', category. This also makes responding easier and less 
time consuming, ■ ; 

G. Sending a reminder post card to reviewers\if they haven 't responded 
after about 3-4 weeks; (Two states have attempted tkas =schnique, 
one attaining- success with it and the other dropping :ne practice 
af ter . some time) , - 

H. Initiating more personal contact with employers /by inannteining -/ 
\ contacts at. the area Chambers of Commerce or at/endlni: trade shows 

\ and collecting business cards, 

I. Encouraging counselors,- teachers, and: others attending User Services 
•..workshops to provide names of potential reviewers. 
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Other suggested improvements to consider include: 

A. Sending a thank you letter : reviewers who have responded, 

B. Issuing certificates to thr "wers who have responded, ^ 

C. Constructing alternative qi .aires for certain types 
of occupations (e.g. profei s) ic allow for more 
specific questions, 

D. ^ Adding certain users (e.g. counselors, tochers) to the 

review panel. Using a different cover letter and questionnaire, 
this group of ' reviewers. mainly would be asked to review items 
such as the format and the readability, rather than the text, 
of the files, 

E. Using colored envelopes to send the questionnaires. 
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SUMMARY 

Verifying information in the system's occupational files and supple- 
menting existing data sources are the major reasons for implementing a 
review panel. To achieve those objectives and to operate ..the process effi- 
ciently requires organized procedures. However, review panels have been 
in existence far too short a time to suggest one proper approach. The steps 
outlined in this paper involve selecting reviewers, telephoning, mailing, 
and recording appropriate details and budget data. A basic outline has been* 
described, along with alternative methods when these are .practiced by the j 
seven states currently operating review panels. Activities required in .j 
the areas of materials development, scheduling, staffing and budget leave j 
ropm for variation. That is, the review panel process remains flexible and/ 
can be adapted to the needs of any career information system. / 
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APPENDIX A 
REVIEW PANEL SCHEDULE 1980V81 



August 
September 

October 

November 

December 
January 

February 

March 



April 



Cluster Number 

8 ICO' 
8400 

4100 
2100 
4300 
2600 

1600! 
3100 
8600' 

5900 
6100 
4700 

5400 
1100 

2400 
2300 
7800 
1400 

1400 
3400 
9400 
4500 

4500 
5600 
7100 
4400 

4200 
7400 
9800 



Cluster N ame \ 

Health Service 
Social Service 



Agriculture, Forestry, & Fishery 
Social Research 
Food Processing 
Science and Laboratory 

Bookkeeping and Accounting 
Mechanic s 
Personal Care 

Other Production 
Transportation 
Graphic Arts 

Metalworking 
Administrative 

Engineering and Design 
Mathematical and Computing 
Food, Service / 
Clerical - 

Clerical 

Building Maintenance 
Protective Service 
Timber Products 

Timber Products 
Electricity and Electronics 
Stock Control 
Textile and Apparel 

Construction 
Salves 

Art and Entertainment 



Number of Occupations' 



19 
16 

10 

' 4' 
4 
12 

10 
16 

,2 

10 
14 

3 

14 
14 

8 
4 
8 
7 

10 

4 

5 
10 

3 
5 
8 
7 

11 
9 
4 
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Ju Y Aug Sepf Oct Nov / Dec Jan Feb Mar Apr May June 



REVIEW PANEL 



REVIEW CYCLE 
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APPENDIX B 

\ . • . 

Here is an example of a telephone conversation to prospective reviewers? 

- Hello. 

- My name is ; , i' m calling from the Career Infor- 
mation System; located in Eugene, 

- I would like to speak to someone in charge of hiring (occupation) . (De- 
pending on the. occupation this could be a manager, supervisor, foreman or 
personnel officer). After being connected with the appropriate person, 
another introduction is. usually necessary followed by something like: 

- CIS collects occupational and educational information and distributes 
it to high schools, community colleges and social service agencies, to 
help people make career decisions. Would you be willing to read a one- 
page description of (occupation) and fill out a short questionnaire? 
Verify the spelling of their name, . their street address and zip code. 
(Optional if they question more about what we are doing or want to x know 
more about us) : ! / 

r CIS is a nonprofit agency that services over (number) of people in the 
state, by providing information to help pl&n their careers. We ask 
people, such as yourself, who are knowledgeable about specific occupations, 
to help us keep our information accurate and up-to^-date by reading and 
commenting on our descriptions, ' 

- Additional .information if they ask: CIS. is a consortium of public agencies 
members of the Board of directors come from the Department" of Highei: Edu- 
cation, the State Department s of Education, The State Employment Division, 
school districts, and community colleges, 

t ■ . - / 

*' ■ \ «=■ ' > 
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CLUSTER: 



CODE 



NAME, ADDRESS .PHONE JITLE 
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AREA SENT 



DATE SENT 



DATE RET. 



UNDEL 
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APPENDIX D 



REVIEW PANEL BUDGET FORM 



Cluster 



Date 



Xeroxing 



// of xeroxed copies made of descriptions 
other xeroxing (describe) 

Total xeroxing 



(§ = $•05 = 
@ $.05 * 



Quick Copy 

// of questionnaires 
// of letters 

Supplies 

// of envelopes iised 

// of return envelopes used 

// of manila envelopes used 

other supplies over $3.00 (describe) 

Total Supplies 



@ $.03 
@ $.03 

« $. ' 
@ $ 



Postage 



postage for cluster 

was this all on postage meter & 
coded "rev"! & cluster // 



yes 
no 



of return envelopes received since 



Total Postage 



Payroll 

// of work study hours for cluster 



@ $.17 
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